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1.0 PRODUCT OVERVIEW 

1.1 Product Description 

This project Is to develop a "Cowaon File System" for 
T0PS2D. The Coinnion File System caDabilities are applicable 
to conf iqurations of two or more 36-bit processors* each 
with its oun main laefflory^ interconnected by a niqh soeed bus 
("CI"). 

The objective of the comroon File System ("CFS") is that 
disk structures and files within such a systeas are available 
to lobs on all processors, reqardless of the physical 
connections of the disk devices. 



1.2 Markets 

The coromon File System is a general operating svstew 
capability and is apnlicable to all present DFCSYSf R*i-20 
markets. 



1,3 CompetitiV)^ Analysis 

This project provides aiore and larqer conf iquration 
alternatives. I his project is not closely related to 
"distributed processing" in that It is only applicable to 
conf iqurations on a CI and therefore within the 100 meter 
limit of the CI. 

VftX/VMS is developing weans for wultiole processors to 
reference files on a sinqle disk system? hOMever, the basic 
difference in ftlesystem architecture between TapS20 and VMS 
makes the projects somewhat different. 

pelated capabilities include "Network file Access" or 
other techniques for wovino files aroonq nodes of a network, 
CFS Is a more powerful and transparent forw of file access 
because it imolegiarits ail monitor file Drimitives visible to 
the user prooraro and operates over a hlqh soeed bus. 

"Multiple Processors" (imolyina shared memory as with 
fOPSlO SMP) is a related capability. Swp is a more po«erful 
approach to the use of multiple processors In that it 
provides qreater transparency and better dynamic load 
levelinq. Jupiter aill not support shared main memory 
however, so an SMP implewentatioa is not possible. Tfiere 
are cowoen<!atinqf qdvantaoes of CFS ovt S«p m the are^ of 
failsoft and isolation of failures, and in the raaximuffl size 
of conf iqurations which can be supported. 
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1.4 Product Audience 

There are two Drlmary types o£ custoroers that will 
require CFS: 

1. Existing customers with KLlO-basad configurations who 
«ish to upgrade to Jupiter and retain their s'assbus 
peripherals. These customers may add a Jupiter to the 
existing conf iauration, and both orocessors will be able to 
access the massbus disks. If additional disk storage is 
provided via eSC-50, the KLIO as well as the Juoiter 
processor ylii have access to it. 

2, Customers who require multiple processors for 
capacity or reliability reasons. The CFS allovs Multiple 
processors to onerate i.«ith a slnqie loqicai disk file system 
such that the iobs may be run on one orocessor or another as 

dictated by load or other considerations. ^^focessors may be 
added to or taken from the conf iquration (subject to 
hardware restrictions) with the remainini? processors 
continuinq to use the file system. 



2.0 PRODUCT aOkLS 
2,1. P PT fornPiViCP 

1. All unpriviieaed woiutor calls which affect disk 
files on oresent one-processor TOPS20 systems «ill work and 
Hill have the intended effect on any disk structure within 

the conf igur atioii. 

2. The overhead associated «ith maintaining the common 
file data base on multiple processors will cause an increase 
of not more than 10% in execution time of file primitives 
and operations. 

3. ft processor referencina files on a disk not directly 
connected will incur no additional overhead in transferring 
data. 

We exoect to use «SCP (Mass Storage Control Protocol) for 
the data transfers to suoport file operations. This will 
exist on the Ci alonq with DECMET and oossibly other 
protocols supDortinq other functions. mscP should achieve 
efficient use of the CI^ low-overhead operation of the 
aionltors/ and hlqh-bandwidth file interchanqe. File 
structure inforsfsation such as directories and index blocks 
is passed exactly as read from disk. By PdS.siaq TOPS 20 file 
data directly, we avoid the overhead of copylno end 
conversion incurred with other nrotocols. 
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Obviously* a processor acting as a file server for 
another processor will incur overhead for tnis activity not 
relatinq to jobs runninq on it. This overhead will involve 
primarilv instructions executed at interruot level and main 
raewory space to buffer data beincr transferred. 

CFS supports shared-Mrltable pages (simultaneous write 
file access) on multiple processors. This is used for 
various internal !!iechanisas (e.q. directory lookup^ disk 
allocation tables) as aell as user prograai functions. This 
type of access generates 10 activity and overhead not 
present on single-processor systems. Because data cannot 
actually be referenced siwultaneouslv by two processors^ it 
must be woved trow one to another by the operatinq system. 
Users «ill be advised of this and should arrange 
applications so as to avoid frequent write references to the 
same data froa different orocessors. 

Since the monitor itself uses this facility,, me conducted 
a study of monitor reference patterns to ensure that this 
activity will not be a significant bottleneck. we recorded 
fBonitor reference patters to directories and disk allocation 
tables under actual and simulated loads. This was done by 
usinq thft SHO'TP f?*cilitv to detp-rt snd record refer once «t 
where the lob maKlnq the reference is different than the Job 
which made the (UOst recent previous reference. This 
orovldes worst-case data on the freauency of moving monitor 
data between processors. We deterrained that only a few <3 
or less) directories were referenced sufiiclently often to 
be of interest. These were all coMmon system directories 
(e.q. <SUBSY3>)/ and the frequency was not so hlqh as to 
suggest a problera. Even so/ a further efficiency is 
possible by duplicating the directory on a separate 
structure for each processor. 



2.2 Environments 

Mlnirouiii conf iquration requires two processors, either 
KLIO^ Jupiter* or mixed* and a CI. Each processor must have 
a connection to the ci. (The question has been raised as to 
whether CFS sight be operable over an NX connection. This 
will not be suooorted In first release. There should be no 
logical reason that HI couldn't be used* but additional 
study and experience is necessary to understand the 
performance implications. Additional imolementation work 
would also be needed.) 

"ach orocessor Must have its os^n main wemorv, swaooinq 
device* hoot device* and console. 

Each processor nsust nave direct access to its oublic disk 
structure. in a future release* it may be possible to 
eliminate this reauireraent if desired* however* there are 
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other requirements for a directly connected disk (e.g. 
swapoinq) which will also have to be addressed. 

The maximum conf iquration for first release of CFS is 
four processors, however there shall be no CFS-specific 
software limitation on a larger number. This iiasit is based 
on our current knowledqe of the CI and the lack of 
experience «ith this architecture. The practical limit way 
be hlqher. A waxiauB of one (dual rail) CI «iil be 
supported for first release. 



2.3 Reliability Goals 

1. A customer should be able to improve net system 
availability of his confiquration bv use of multiple 

processors and tiie CFS. 

2. The CFS should cause no slonificant decrease in the 
reliability of each siaqle processor. 

3. Failure oi one processor will have no effect on other 
processors ewceot for file dat^a which is in th*> fPfBorv of 
the failinq processor, 

KL-Jur>lter conf i quratlors such as this r^rovide some 
additional redundancy over sinqle-orocessor systems. The 
fail-sott characteristics are not synssnetric nowever. The 
Jupiter processor and/or its attached disk can fail without 
affectina lobs on the KL processor or files on the KL dislrs. 
Hoyever, failure of the KL may have mora impact dependinq on 
what Peripherals are connected to the Jupiter. 



2.4 Mon Goals 

1. CFS will not suPDort other than disk structures. 

2. CFS is not intended to Mork with operatinq systems 
other than TQPSIQ or «ith nachina architectures other than 
36-bit. 

3. At first release^ CFS ^ill not support passinq file 
data throuqh one processor to another except for the case of 
a KLIO processor actinq as file server for asassbus disks. 
That is, a Juoiter processor will never act as a 
pass-throuqh for file data, Handlinq pass through data 
increases overheau and frtlsHs the oossibilitv of roultiole 
paths between a processor and a -Hsk which ^lould reauire 
routinq decisions. CFS asay be used in configurations with 
multinle CT's, but such conf iqurations mav have processors 
that are unable to cofsuiunicate with soine disks. These 
restrictions may be removed in a subsequent release. 
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4. CFS does not provide any automatic baiancinq of lob 
load amonq orocessors ir> a conf iquration as does swp. 
However^, users should find it convenient to login to the 
less loaded orocessor and/or to switch processors (by 
logging out and back in again) If the load becomes 
unbalanced. 



2.5 Future Deveiopwent 

In a future release^ it may be desirable to allow Jupiter 
processors to pass file data so that configurations roay be 
built in yhich some processors do not have direct access to 
some disks. in qeneralr this will also reauire routing 
decisions to be made since multiple paths can easily arise 
once there are several CI's in a configuration. It would 
still be avisable to }i)iniroi?e the amount of oass-throuqh 
traffic in order to ainimize overhead, but pass-through 
offers sizable configuration flexibility and some failsoft 
features. 



5.0 FUNCTIONAL DEFINITTON 

3.1 Operational Description 

Two or fflore processors are interconnected via a 
high-speed bus ("CI") having a bandwidth at least comparable 
to disk transfer rates. A disic which is to be used by a 
processor must have a direct path to that processor, i.e. 
be on the sane CI. The only supported exception to this 
Mill be to alioM files to pass through the KLIO to reach 
Jupiter so that tiles on Massbus devices may be used by 
Jupiter processors. 
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One or more loaical structures exist oa the set of disks. 

Ill of these structures r»ro visible to loos on all of the 

processors unless the system adKiinistrator specificailv 

declares particular structures as 'orivate' to particular 
processors. 

In order to provide access to Massbus disks connected to 
a KLIO, the KLIO «ill act as a logical disk controller on 
the CI for the Massbus disks. There is no visible 
distinction between a disk structure directly connected to a 
processor and one which is accessed via another processor. 
The usual monitor calls are used to access files and 
structures^ and ali file open modes are allowed with the 
exceptions listed below. Shared file access is permttted, 
and programs need not be aware that other jobs sharing a 
file are on different processors; however, it may is 
advisable for reasons of efficiency to avoid simultaneous 
modification of a file on different processors. 

in future releases, it may be desirable to support 
configurations which include multiple CI 's, each «ith a 
subset of the orocessors connected. In such cases, it way 
be desirable for Juolter CPUs to act as pass-throuqh servers 
In order to provide access to a disk from another processor. 
This will not be suoporteci st first release. 



File facilities soeciticallv include: 
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1. File naminq and lookup conventioiis (GTJFN) 
File names on the coatnon file system include structur€# 
directory and subdirectories^ file name# extension^ 
aeneration nunaberr and attributes. Full recoqnltion 
and wild-cardinq Is available; name steppinq (G^■JFW)? 
normal access to FDR. 

2. Usual open and close modes (OPENFr CLOSf# 
CLZFP). 

3. Usual data transfer primiti^res, both sequential 
and random (BIM^ BOUT, SIM, SQOT^ SIMR, SOUTR, RIM^ 
ROUT# DIWPI, OUMPO). 

4. File-to-process jsappinq <PNAP) includmq ail 
modes (shared read, copy-on-«rite# shared write, 
unrestricted read). 

5. The device type associated with files on the 
common file systeff. Is the same as that presently used 
for disk. 

6. Privileqed operations MSTR (mount structure) and 

nsfnp . 

The above incluaes all file systefii orionitives relatinq to 
accessinq files and transferrlna data but does not Include 
other primitives wnich mav use certain file system entities 
but which are considered separate and distinct facilities 
(e.q. EMO/DEQ). 

See J^ppendix 1 for impleaientation considerations. 



3.2 Restrictions 

A file open «ith OFtDUD (don't update disk) on one 
processor may not be opened on any other processor. 

Other devices such as niaqtapes and line printers are not 
part of CFS and aav not be open simultaneously on multiple 
processors. 

Use of simultaneous write access with active wrltinq of 
file data by jobs on different processors requires the 
systeai to move pages aaonq the processors and hence will be 
much slower than on a sinale processor. The -»»rite token is 
maintained on a per-QFS basis. This means that a proqraa 
requiriaq urite access to any one or wore paoes must nave 
exclu5iv*> access to the entire 0F«^. ?ach 0F« reoresents 
256K i*ords of tne file. For larqe files, proqrams on 
different processors could be executino simultaneous write 
references with no delay if thev were referencina data in 
different 256K sections of the file. 
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ft structure j^ust be "aounted" on any processor which is 
to access files on it. To be ohysically removed from a 
drive, a structure Riust be dismounted by all orocessors. 
The relevant Galaxy coroportents should be modified to provide 
mount information froro processors other than the one on 
which they are runninq^ but this in not planned for Jupiter 
CFS. Hence an operator will have to auery the npR oroaraB 
on each nrocessor to find out «hat users have the structure 
counted. Each processor will know, ho«evei> which other 
processors have the structure aounted so that the operator 
can quickly determine if the structure can be removed. 

Dual port ooeration of a single drive froia a single KHO 
is not currently suooorted. If it is implefflented^ the fact 
that there are tyo oaths to a disk will not be visible to 
CFS and will have no impact on the exclusion of routinq 
capabilities in the first release. 



4.0 CDMPATIBILIIY 

4.1 DEC Products 

411 oroaram and user interfaces ^re coaoatible with 
previous versions of TnPS20, 

Mountable disK structures are coinDatibie adth previous 
versions of TOPS20. 



4.2 DEC Standards 

The CFS will use the corporate SCA protocol on the ICCS 
bus# and will use MSCP for disk data transfer operations. 

The CFS will not use DECNET. 



4.3 External Standards 
^one aoolicabie. 
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5.0 EXTERNAL INTEHACTIOKS AMD IMPACT 

5.1 Users 

All users of the disk file system are potential users of 
CFS, however fflost users will not be aware of or affected bv 
CFS. So»e apnlications developers will rely on CFS to allow 
applications to exist on Bultiple processors and cowfflunlcate 
throuqh files. 

Other proiects yithln the Coexistence and Distributed 
Processing group will use CFS to provide layered 
functionality on aultiple homogeneous processors. 



5.2 Software Products 

5.3 Products That Use This Product 

The following may use CFSi RMS, Dl¥, languaoe OTS's, 

5.4 Products That This Product Uses 

The fnlloi;ing hardware co'^irsonents ar^^ required: 

KLIPA - Interlace oet^een KLIO and ICC3 bus. 

Jupiter CI Port - interface between Jupiter and ICCS 

bus. 

KLIO Microcode - modifications to support "write 
access in CST". 

The followintj software modules are required: 

«SCP driver and server in TOPS20. 

Systems coiainunications Services (SCA/SCS). 

5.5 Other Systems (Networks) 

The CFS Is not visible to other network hosts? the files 

in the CFS dis*r structures way be accessable by remote nodes 
as provided by other facilities iDkP, H¥T, etc.) Each 
processor in ,i CFo conf i qur a t i on 1;=^ -i seodrate network noue 
with its own node name. 

The CFS Itself does not vsp. node names to reference files 
and hence is inaependent of any constraints or requirements 
of network node namlna. 
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5.6 Data Storaqe, File Structures^ Data For.i^ats 

and Retrieval Subsystem 

The CFS requires an open file data base wnlch is resident 
In each Drocessor of a configuration. 2-4 words oer OFN are 
requlre<1. Other resident storaqe reaulreu^eitts are one paqe 
(512 words) or less. As a side ef tect oi allouinq all 
processors access to all counted structures^ it may be 
desirable to build standard monitors with a iarqer number of 
countable structures than at present. 

The file structure Hill be Identical aith previous 
releases of TOPS20. 

Files way be saved and restored «lth DUMPER without 
reqard to whicn orocassor OU*?pER is run on, except that 
DUWPER ^ust be running on the processor uhich has direct 
connection to the required tape drive. 



5.7 Protocols 

^FS yill use the corporate S€A protocol on the ICSS bus 

and will use '-^SCP for data transfer. 

CFS will use a private protocol tor control of file 
openings, structure jiiounts, file state tiaasitions, etc. 
There is no present corporate protocol «hich supports these 
functions. 



6.0 RELIASlLlTY/AyAILABILITV/SERVICEilSILITy (HAS) 

6.1 Failures within The Product 

Failures within the CFS-specific softyare will roost 
likely cause a crash of one processor in a multi-orocessor 
environwent. Such failures may include loss of recently 
modified file data. Failures Mhich affect inactive files or 
file directories are possible^ but should be no more 
freauent than at oresent. 
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k processor may be broijqht on line without restartlna 
other processors in the configuration. 

Any disks «hich are available only via a tailed processor 
will be unavailable so lonq as that processor is 
inoperative. If such disks are dual-ported to a different 
processor^ thev may be mounted via that processor and reaain 
In use although all open files must be re-opened, 

With HSC50, most disk errors will not be seen by the 
processorCs). All recovery and loqqinq »'lil be handled by 
the HSC50. Any disk errors that are reported to the 
processor will be logged in the systeiu error file for that 
processor. Disk errors occurinq on pages that are beinq 
"passed throuqh" a processor (e,q. a KLIO servlcinq a 
request for a Massbus disk) will be loqqed on the processor 
to which the disk Is directly connected, I£ a hard failure 
occurs such that the server processor must inform the 
requestinq processor that the request could not be 
completed, then the reauestinq processor will also loq the 
failure. 



7.0 PilCKAGlMH AND SVSTS^ nENESflTlON 

7.1 Distribution Media 

CFS will be an integral oart of TQPS20 and will require 
no additional aackaqinq. 



7,2 Sysqen Procedures 

CFS code aill be in all ntonitors. No specific monitor 
builds are required. 
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9.0 DCCUMEJiTATIu?! 

installation Guide - Describe possible ci="S 
conf Iqurations; describe installation of CFS, See also 
"Installation Procedure" Functional Soec. 

System AdfBinistrator's Guide - Describe characteristics 
of CFS? assignment of iobs to processors. 

Monitor Calls Reference Manual: JS¥S to control 
configuration (siaall chanqe to mstr%)j error codes. 

SVSERP Manual - Specific error cases. 

Operator's Guide - Bringing individual processors up and 
dovn. 



9.0 RBFEREMCFS 

1. Functional Specification for Loosely Couoled Systems 
(LCS) - Fred Engel, 30 April IQRO 

2. Jupiter Conf lauration Recommendation (Memo) - Peter 
Hurley, 1 May 1.^81 

3. LCS and tiie Conijaon File SysteiO (Memo) - Daii Murphy, 
15 Jan 1980 



10.0 APPENDIX 

10.1 Irsoleeientation Details 

All transfers «111 be in paae (512-Hord) units. The 
PAGEM-PHYSIO interface Mill recognize a request that must be 
handled via the CI and another processor instead of a direct 
disk transfer. That is, the access to the common file 
systeai disks will oe exclusivelv through address inappina as 
it is now. If the full ci hardware ware available, a disk 
transfer could always be requested directly and moved to or 
from siGBiorv without the Involvement of any other processor, 
in actual systems (including KLlOs), a particular processor 
raay not have direct access to a particular disk and so must 
foryard a request to a processor which does. The processor 
laakina the request acts in exactly the same «av as if the 
request had been made to disk; when the page is ultiaatelv 
received from the other processor, th« usuaj transfer 
cowoletion action is invoked. Some processors must be 
'servers' in such a system. A server must oe prepared to 
receive requests for disk transfers from other processors at 
any tlwe and respond to them quickly. This is reasonably 
simple; the server processor places the dtsK request in its 
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own disk transfer queues ( includinq allocatinq so«e wemory) 
and when the transfer is con'olete, starts another transfer 
of the data over the processor- to-orocessor link to the 
destination orocessor. 

We intend to provide this by implementinq an MSCP server 
module. Thus/ a host processor will make disk requests in 
the same way whether the disk is directly connected via the 
CI or indirectly accessed via another processor. 



Open File Data Base 

A fflajor new element of the common file system is the 

common open file data base. This is sowewhat equivalent to 

the present OF»l data base in TOPS20 but is maintained 
iolntly on all processors. Interlocks and updating 

mechanisms serve to let each processor maka necessary 

accesses to Its o«n copy of the data, to keep all copies 

updated, and to haiulle a processor crashing and coininq back 
on line. 

Files that are ooen exclusively on one processor or on 
•tultiole ornr** «?sf>r*' for roj^d-nnlv finrludina cmy-o^-wri te) 
can be handled witn no delays except for the data base lock. 
Simultaneous write on multiple orocessors requires some 
additional alqoritnT!»5# primarily the tiasslnq of the "Hrlte 
token" troffl processor to processor. There is a separate 
write token for each open file, Tne write totcen ioqically 
exists on only one processor at a tine and allows orocesses 
running on that processor to read and write data in the file 
to which the write token applies. If a processor does not 
have the «rite token for a file# no accesses* read or write* 
are allowed. The write token for a file which is 
shared-writable and is being actively used by multiple 
processors will be rotated asRonq the processors. If only 
one processor is demanding it* it will move to that 
processor after a short tipie interval to prevent thrashmq. 
Also* a maxiffluju time limit Kill be used to prevent any one 
processor frora hoqqinq in the oresence of demand froro 
others. 

in order to q1v« up the write token* a processor must 
make InaccessihlA i^^ll of the naaes of the file to which it 
applies. This is done bv jnodifyinq the index block or SPT 
for paqes then m use and writinq any modified paqes to 
their home addresses on disk. 

Since all orocessors are runnina the same operatinq 
system, we assmae a cotisistent alyorithn? lor requestiriq and 
oivino uo the write token will be oresent on all. The rsT 
will be used to detect the first Hrlt<=( reference to a paqe 
so that the processor can request the write token for the 
file if it Is not alreadv owned. 
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'^e have consiaered havlna the write token exist on a 
slnqle-paqe basis^ however the data base reaulred to support 
this seems to be too areat. On the other hand, it Is 
possible to leave file paqes readable when the file write 
token passes to another processor and then only invalidate 
pages when the processor with the write token actually 
modifies them. This incurs a delay each tisiie a paqe is 
first accessed for write however, since the writlnq nrocpss 
must «ait until the invalidate is acknowledged bv all other 
processors. 

There Is a file opening wode, QF%RD{J, ahich always is 
allowed but does not auarantee that the file will be seen in 
a consistent state, A file open in this state on one 
processor way continue to be read when another processor has 
the write token. 



Directories 

Once we assume the basic file maooinq mechanism as above, 
most of the remainder of the TOPS20 file system works 
without significant modification. Directories are mapped 
into the process address soac<=> ^nH rof or^^nced by the rrocess 
as at present, A common inter-orocessor lock mechanism must 
be iB^ple-nented to handle directory locks. The write- token 
mechanlsf* will serve to allow each processor to niodify the 
directory as necessary. Most directories are referenced 
only by a single job and so should present no problem of 
excess movement of the write token. Some commor, directories 
(e.g. <SUBSYS>) are referenced frequently by manv jobs but 
only for read; so again delays for write-token latency 
should not be a problem. 

The disk allocation tables are raaooed in the same wav a«? 
directories, and the basic page fflechanlsw will support this. 
There is already a explicit lock on the use of these tables? 
making it global will prevent any innroper simultaneous 
reference by multiple processors. We are planning a study 
of the patterns of use of the allocation tables to deteriulne 
If there will be excessive traffic between processors. 
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Structure Data Base 

The structure data base yill remain aenerally as it is 
now. All structures which are accessaole ("spinninq") will 
be Included, whether or not anv users on the system have 
mounted thee. When the number of users havinq a structure 
mounted on a particular processor changes frow to non-0 or 
vice versa, a raessaqe is broadcast to all other processors* 
Each processor, on receipt of such a messaae^ adjusts its 
oiin data to reflect which other processors have the 
structure Mounted, This can be shown as a "funny" user 
naffie, e.q. *KL2102* so that moU>ITR will knoy whether the 
structure can be physically removed from the drive. 

The structure data base will also record if a structure 
has been declared "private" to one processor and will iiwit 
access accordlnqiy. 



MftJQR AREAS OF DEVELOPMENT - Effected modules 



T. ""lie System Initialization {FILI*JI, miEX^^C) 

Exchanqe conf Iquratlon Informatlan? each nrocessor 
knows of all others. 

Exchange mounted structure info. 

II. ^'ount Structure (MSTR) 

III. Handle disk allocation tables, (DSKALC) 

Global (inter-processor) lock set while modifications 
In proqress. Make present lock: qlobal. Allow bit 
table routines to run OKSK£D. 

Write-token mechanism handles concurrent modification 
by Kultiole orocessors. 

IV, own (DISC, PAGEM) 

Wrlte-token handled on ner-OFN b^sis (equivalent to 
per-file for non-lonq files). 

Finite state wiodel to renresent movement of 
write-token, undatina of paqes, etc. 

Stipoort of sItBultanftOiJfv ^rite for OFtDUD requires extra 
worst - may be deferred. (I.e., limitation is that all 
openers of LK*onD tile must be on sai'ie processor, > 

V, Paae Manaqenient (APRSRV, CISRV, LCS3RV, PAGEI^, SCHEO) 
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Detect first write to oaqe (bit in CST - ucode 
requireraent). 

Force out ail paqes in QFU, assuwinq none modified. 

Detect reference to paqe in OFN not currently 
available. 

Transfer page to/from other systems. 

VI. >^isc (BUGS, GLOBS^ STG) 

OFN STATES 

1. Open/ any processor read 

10. Exclusive request sent, aaitlnq for response. 

2. Open, exclusive not self (i.e. access prohibited) 

20. Share request sent, waitinq for response. 

21. Wait before repeat send of share request. 

3. Not in use 

30. Status request sent, waitinq for response. 

4. Exclusivtj, self 

Reoue'^t conflict resolution: 

While waiting for response to a sent request, a 
conflictinq request ^ay oe received. Based on the 
types of the two requests, one of tyo tnings is done: 

1. Deny the received request, continue to wait 
for response to the sent request. 

2, Grant the received request, continue to wait 
for denial of sent request. 

The priority of requests are: 

1. Exclusive 

2. Share 

Each processor is assiqned an arbitrary priority which 
is assumed to have no slqniflcant affect on system 
performance. 

The rules are; 

If received request is higher priority than sent 
request, then do 2j 

If receiv«»d request i «; }o4fc orioritv, then do 1* 

Itherwisa, if tiiis orocessor has hioher oriorlty, 

do Ij Otherwise, do 2, 
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Any received request which is compatible with the current 
state is aranted, even if no chanae of state occurs in the 
local OFN, Hence, orocessors do not haye to change states 
of OFf^'s not belnq actively used. 

Note the above is only to resolve race conditions. Once the 
race is resolved, the "losing" processor will nait for a 
small tim& and then resend the request. Exclusive or shared 
access must be given up unon request if the access has been 
available for longer tnan a specified period of time. (This 
period may be dynaroicailv deterniined based on number of 
pages in use or nsodified, i.e. ^ork necessary to give up 
access, ) 



CFS3RV is a lock manager. The locks it iaanaqes reoresent resources 
in the system, but CFSSRV is not aware of the mapoino of lock 
to resource. The inappinq^ or meaning* is wade by the creator of 
the resource. 

Files are a resource with CFS locks. Each file has the followinq 
CFS locks: 

• open tyoe 

• write access 

. ENO/DEG lock 

in addition, each of the sections of the file, represented by an OFN# 
has an access token. Therefore a file has up to 512 access tokens. 

When a file is opened, the "open type" and "«*ritfcj access" lock are 
acquired. The "open type" is either" 

. shared read f frozen) 

. shared read/write (thawed) 

. exclusive <restricted) 

. Dro^i i^cuoti?; (unre^^tri rtad) 

The word in pareiitheses represents tne arqument to OPLnF%. 

If the opener requests "trozen write" access, then If the 
"open type" loc'< is successfully locked, i.e. no one has the 
file open in a conflictlna liode, the "write access" lock is 
acquired. This is an exclusive lock that represents the 
single "frozen «rite" user of the file. The lock is held by 
the systea that nas the file opened "frozen wirite". 

Each of the locks described above aooiy to a file, that is 
soraethina descrioed by an FOB. In addition to these, each 
file has soaie nuwber of OFMs, one for each file section that 
is in use. Therefore, a file may have up to 512 OFNs or file 
sections. 

Each active OFN has an "access token" lock. The access token 
represents the ability of the system to access the data 
described by the OFN. The access token may be held In one 
of the folloylng modes: 

, olace-holder 
. read-only 

. exclusive (read or write) 

A read-only access token roav he held by any number of systems simultaneously. 
An exclusive token is held by only one system, k "olace-holder" 
access token is an artifact that nerTits the CFS systews to aares 



on the end-of-file correctly. It also has soae ratifications for 
bit table access tokens that will be described later. Place-holder 
tokens are also an optimization to avoid reallocating tokens that have 
been "lost" to another systew. 

The file access token is the most fundamental CFS lock in that 
it is used not only to corstrol simultaneous access to user files^ 
but also to manage directories and bit tables. 

The access token state transition table is given below, with the 
action required to nake the designated state change 

exclusive place-holder 
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read 


read 


nothing 


exclusive 


DDM?** 


place-holder 


vote 


Where: 





vote DDHP* 

nothing DDMP* 
vote nothing 



vote weans that the other C^^^ svstews must b*» askpid for Der^ls«!ion 

to make the state transition, Noting is a fundamental operation 
of CFSSRV and is done bv a softyare implemented broadcast. 

DDMP* ffleaiis thdt DDMP aust run and remove all of the OFK's pages 
from memory and update the disk cooy oi any modified pages. 

DDMP** means that DDMP must run to update to disk any modified 

pages and any in ooeiBory pages must be set to "read only". 

This latter operation is oerfor^Bed by clearing the CST write 

bit. The CST write bit has been implemented in KL paging explicitly 

to support loosely-coupled nulti-processors. 

While DDMP is oerfoming a CFS-directed operation^ all pages of the 
OFN are inaccessible to aiiv other process. Tnis is achieved by 
a bit, SPTFO, set in SPT02 bv DDMP. 

iccess permission to a file moves among the CFS systems or? demand, fach systeii 
must remember its state of the token so it may respond to requests 
for the access oerfflission. 

The token consists of? 

, The structure narae 

. the OFN disk address 

, a flag bit to Indicate this is the access token 

. state 

. end-of-fiie pointer 

. end-of-file transaction number 



. fairness tiroer 

. the OFw this token is for 

and, if this is a token for a bit table? 

, structure free count 

• structure free count transaction nuisoer 

The fairness timer is a cfS service that allows a resource to be held 
on a node for a guaranteed interval. Therefore^ the owner need not 
lock the resource and arranae to unlock it later. Father it slraply 
places the guarantee iriterval in the resource block and the 
CFS orotocol takes care of the rest. 

Place-holder tokens exist orincipally to hold the values associated 
aith the end-of-file pointer and with the structure free count. It is 
Important that these be held by each system, because the owner of 
the OFN token may crash and therefore the last known state of these 
quantities must be remembered so that the reaaininq nodes may have 
the best possible value for thew. The transaction count is intended to 
determine whose value is the most recent snould the owner not 
be Present to contribute the current value. Durlnq the votinq 
for acqulrlnq a token, these values are passed amonq the CFS nodes# 

and the "f»<i*» f-onrjlict jno th*> vnfp ria+siin'; fhp vj}up^ 3ic;ir rii^^-j •»+ orf '^it^l 

the largest transaction number. 

The file access token represents the riqhts that a system has to access 
a file section. That is, the tojcen is associated with the file's contents. 

However, the owner of a file, i.e. the system holding exclusive rights 
to access the file, also has the riaht to modify the file's index 

block. The owninc} system may add pages to the iile or delete pages froa 
the file. 

OFHs are treated specially in TOPS-20. Unlike the file's data pages, 

an OFN may not be discarded when the system gives up its access 

to the file and read from its ho»e on the disk when the access 

is reacquired. An active index block, represented by an OFM, contains 

paging information that must be retained while the file is opened. 

For this reason# a systero needs to be informed If the index block 

contents are changed by another system. 

This information is disseminated in CF3 by a broadcast message. Each 

tlwe a svsteffl writes a changed index block to disk, it Inforw? all 

of the other C*^S systeips by a broadcast message. Note that this broadcasting 

Is done only when the changed index block is written to disk, and not 

each time the index block is modified, a broadcast message Is used 

Instead of including this in optional data with the access token 

for reasons explained in a later section of this docusjent. 

When a C^S system r*»ceiv*>F "^uch a "'?»s<^3ne, it s^ts ? <;tatt)s bit Ir 

the aoporpriate UFU so that the next time a process attemots to reference 

the CFN the foliowmq yill happen: 

. the disk copy of the index block is examined. 
. for each changed entrv/ updats tne local OFN 



This reconciliation of the index block aith the local OFN is accoraplished 
by the routine DDX8I. 

VOTIMG in CFS 

When a node needs to "uograde" Its access to a resource* including 
acquirinq a new resource* it must uoll each of the other CFS nodes. 
This is so because aorie of the CFS nodes is a aaster and therefore 
there is no a priori location for resolving access requests. CFS 
is not only a democracy* but soroewhat of a cacophony. 

voting* then* requires "broadcastinq" to each other node the 
required resource and access. Each node aust respond with its 
permission or denial. 

The CI does not support broadcast, and even if it did* it would 

not support a reliable broadcast. Therefore* CFS laolements broadcasting 

by sending a wessaqe to each of the other nodes* one-at-a-tirae. 

k vote request contaias: 

• function code 

. resource "name" (seventy-two bits) 

. vote number 

A reply containsi 

. function code 

, resource name Cseventy-tuo bits) 

. reply <yes* no or "qualified yes") 

. vote number 

. optional data 

The message contains a function code because votes and reolies are 
only one kind of CFS to CFS corarounication. 

The vote number is used to insure that the reply is to the proper 
request. The rerruestor may "restart" a vote at ?ny time. It does 
this be "canceling" the current vote* acquiring a new vote number* 
and broadcastinq the new request, k vote number is a monotonically 
increasing* thirty-six bit auantity. 

A vote will be restarted for one of the following reasons: 

, a conf iouration ch?»nqo is r^oorted by SCA 

. a previous vote "tlrries out". 

The latter should rarely occur* and is likely indicative of a 

malfunctioning CFS on some other system. In soBse cases, a node will 

not reply if it is unable to acquire the aooroprlate «?pace for constructing 



a message. There are a small number of cases where this is leaal^ 
and for these cases, the requestor must revote when appropriate. 

When a reply is received^ the vote number must match the number in 
the associated resource block. 

The replies to a vote are: 

. unconditional yes. 

. no 

. conditional yes, 

. cancel yes condition 

ft conditional yes nieans that the respondent will approve the requests but 
it needs to perform a local housekeeoina ooeratlon first. The most connon 
form of this is voting for an access token where the respondent must first 
update the disk copy of the file, and oerhaos flush all of its local 

copies of the file data. When the condition has been satisfied, a "condition 
satisfied" reply is sent. 

Each resource has a "delay wask". This niask has a bit for each of the 
other CFS nodes, and whenever a node reolies with "conditional yes"# 
its bit is set in the resource's delay mask. Tnerefore, a process 
that i *5 waiting ior tho cnndi ♦"ion<^ tn h*^ «;^t ic't i ed/ si'^r'lv examines 
the delay mask periodically and «aits for all of the delay bits 
to be cleared, Wnile any delay bits are set, tae vote is considered 
to be still In progress, and therefore ^nv conf lauratlon chanae 
will require restarting the vote. 

Conditional yes votes, and the associated delay mask, are provided 
to eliminate the need for nodes to reply "no" «hen there are 
temporary conditions preventing the approval of the request. The 
overhead required to process such reolies, and to wait for the®, 
is offset by the gains in not havinq to revote in the face of such 
conditions. 

CFS provides the folloaina basic votinq services: 

. Acquire a resource. If the resource is known on this 
node, but the current state conflicts yith the request, the 
currently held resource is released and a vote is taken. 

This service is called specifyina either "retry until 
successful", or return after one try. 

. Upgrade a resource. This service tries only once. It 
also guarantees that the currently held resource will not 
be released. In fact, the resource may be held and •♦locked" 
locally when "upgrade" is requested, 

. Acquire local resource. This is u«?ed for rs^sources 

not shared by other CFS nodes, but manaaed by CFS. fixawoles 

are directory locks on exclusive structures, 

VOTE MECHANISM 



h vote is started by the routiae VOTEW. Ordiinarlly, one does not 
call this routine directly, but rather one requests a resource^ 
and if necessary, VOTEW will be called to conduct a vote. 

VOTEW always waits for the vote results. The results are tallied 
at interrupt level by noting the number of replies received in 
the associated resource block. VOTEW periodically examines the 
resource block testing for: 

. all tallies received 

, a "no" vote recorded 

. a configuration change 

The actions taken are as followst 

. configuration change: restart the vote 

. a ♦•no" vote; return to the called 

. all tallies received: 

. it no "conditional yes" votes, return to caller 

. If one or more "conditional yes" votes, wait for 
tho "cnnditif^r *5?ti«;if i«»d" raolles. while waiting^ 
a contiquration chanoe could occur, renulrin<i the vote 

to be restarted, 

RESOURCE ACQUISIIIUM AHD UFDATIfiG 

CFS resources are acguired and changed in response to requests 
from other t^arts of the Eonitor. Rather than describe each one, 
it yill be Instructive to consider how the file related resources 
are acguired, maintained, and destroyed. 

When a file is opened, and the first QFU is created, aSOFN will 
create the static CFS resources: open type and, if appropriate, the 
frozen writer token. 

Anytime an OFN is created, be it in response to opening the file, 
or one of the "long file" QFNs, aSQFN mIII create the access token. 

The access token state is verified by various of the file system 
and wemory wanagement routines. The ■nost co-nffon olace for this 
is In the page fault handler. The two exceptions to this are 
for a bit table access token and a long file "super index 
block". The bit table token is acguired and "locxed" when 
the bit table lock is locked 3nd released only when the bit table 
is unlocked. The token for a super index block is occasionally 
acauired in DISC by the routine (^iSWLFT) that creates nea lono 
file index blocks. Tn theorv, the<!e ercaDtion c^ses need not 
be exceptions, fnat is, the code could .simply rely on the 
normal management of the token durina oaae faults to insure 
data integrity. However, In these cases, the code laust oerforw 
multiple operations on the file data "atomicaliy". That is, 
it must modify two or <^ore pages, or it wust "test and set" 
a location '^ith the assurance that no other accesses to the 



data occur between the steps. On a single system, this Is 

done by a NOSKED to prevent any other orocess from runnino. 

In an LCS environment^ MOSKED is not sufficient (althouqh it 

Is necessary!). Another forcn of interlock roust be used to prevent 

a process on another systems from examlninq or modifying the 

data. It turns out that the access token satisfies this 

need quite well. 

The above discussion implies that the oaqe fault handler, Mhen It 
acquires an access token for an QFN# dOBS not "lock" the token on 
the system. That is, the token is acquired but not "held". This 
may result in the token beinq preesjoted by another system before 
the process is able to reexecute the instruction that caused the 
paqe fault. The "fairness" timer in the token resource is one 
attenipt to minifflize such thrashing. 

The access token is acquired on the folloMing conditions: 

. when an QFN is beinq created 

. yhen the jFfJ is locked 

, when a oaqe fault occurs because tne current access is 
not correct 

The current state of the token is kept in the CFS resource block as 

well as in thp Of h f^3t=» h?!<;e» The fi*»^<^, *^PTST,, i«? the rnrr^nt 
OFN state of an OFN. The values are: 

-> no access 

.SPSRD => read only 

.SPS¥R => read/write 

SPT3T is modified by the routines in CFSSRV that are called to set 
the state of the file. The values are set here, and not in PAGEM, 
PAGFIL or PAGUTL because the Q¥U state must be set while the 
CFS resource block is interlocked aqainst change. 

The routines to modify the state of an OFN token are: 

. CFSAWT - acquire token but don't hold it 

. CFSAUP - acquire token and hold it 

TOkEM **«,*)l';E«E?iT 

once a token is "owned" on a system, it will remain in that state until 
it is required on another system. That is, if the token is held 
for read/writa access (exclusive), then all references to the 
paqes of the OFN will succeed without CFSSRV being invoked. 

If a token «ust be revoked because another system needs it, CPS*5RV 
slqnals DDMP to process the data oaqes. This is done by: 

. Setting bits in the field STP3R in tne GFS data base. 

. Setting the DFH's bit in the bit mask OFNCFS. 



. Making up DDMP, 

The field STPSR is a two-bit ouantitv Itidicatinq the type of access 
reauired by the requesting system. DDmP's action is as follows: 

read-only neededi 

Write all modified oaqes to the disk. Clear all of the CST 
write bits in all iTi-memory paaes. 

read/«rite needed: 

Write all modified pages to disk. Flush all "local" copies 
of data includinc) any copies on the swappinq soace. Snap out 
the OFM page if it is in Ti!e?sory (actually, slwoly place it on 

RPLQ). 

Once DDMP has performed the necessary operation, it calls CFSFOD, 
This routine Mill set the OFN state and the resource state 
appropriately as folloiHs: 

read-only requested: 

set OFM state to .SPSHD and set resource state to "read", 
read/yrite requested: 

set OFn state to and set resource state to "place-holder". 

CPSFOn also copies the currf?nt pnd-of-file Inf orti^ation frojn OFNLEN 
into the resource ulock and tlnaiiy it sends tne "condition satisfied" 
message to the requestor. 

While DDMP is performing its work: on behalf of CFS, it sets the 
bit SPTFO in the OFN data base. This bit Is examined by the 
page fault handler, and by CFSAWP/CFSAWT to see if the 
OFN is in a transition state. If SPTFG is set# and the process 
requiring the OFU is not DDMP, then the process is blocked until 
SPTFO is cleared by DDMP. In order to facilitate identifying 
DDMP from all other processes, a new word has been added to the 
PSB called DDPFRK, If DDPFPfC is non-zero, then the current process 
Is indeed DDHP and SPTFO should be ignored. 

USUSED RESOURCES 

Whenever a node replies "no" to a request, it remembers in the 

associated resource bloclr the node(s> that have been rejected. The 

only reason for unconditionally denying a request is that the 

resource is "held" locally, if a resource cannot be granted 

because of the fairness tiroer, the "no" response Includes 

an optional data word of the tlise the resource is to be held. Therefore, 

the requestor knows preciselv «hen to request the resource anew. 

When a held resource is "released" (or undeclared), CFS examines the 
rejection mask for the resource. For each node identified in the wiask, 
a "resource released" message is sent indicating that this is 
a propitious time to try to acquire the resource. There is no 
guarantee the nei* request will be granted as the resource could 
be held again, or another node could have requested, and been 
granted, the resource first. 



DELETING FILE SeSQURCES 

The access token is deleted whenever the associated QFM is deassiqned. 

The static file resources are released «^hen the file is closed. This 
is oerformed in RELQFn. 

CHAMGES TO EXISTING COMCURRENCY CONTROL SCHEMES 

As a result of CFS^ much of the concurrency control in TOPS-20 has 
becofflie distributed. In some cases^ this has Deen done bv creating 
a comoanion resource to an already existing one. As example of 
this Is the file open mode resource described above. 

m other caseSi, existing locks have been replaced bv CFS resources. 

The decision as to which techniaue to employ was made on a case-by-case 

basis. The sionific^jnt criterion was how easy it was to eliwinate 

the existinci concurrency control and replace it with the CFS 

manaqement. The file resources proved difficult to do. However^ 

there are two important pieces of the monitor's structure that 

Mere easily and efficiently replaced: directory locks and 

directory allocation tables, 

nir*»rt-orv }nrk^. ar*> nn'.4 CF^ ro<?<nirrp'=r , h f?-|r^rtrirv 1 nrlr ro^ourCP 
contains: 

, the savanty-t¥C bit identifier 

, owning fonc 

. access type 

, share count 

. waiting fork bit table 

In fact, a directory lock resource is the sole instance of a "CFS 
lonq block". 

Directory locks are always acquired for exclusive use. However, unlike 
file access tokens, directory locks are never <jranted »*conditlonally". 
This is because directories are files, and the directory contents 
are subject to neqotiation by the associated file access token. That is, 
acqulrino exclusive use of the directory lock resource is independent 
of acquiring permission to read or '^rlte the directory contents, When 
soroe process on the oMnlnq system attempts to read or write the 
directory contents, it must first acquire the file access token In the 
proper state, ftlthouoh this sounds someMhat inefficient, i,e, requiring 
the node to acquire two independent resources, it is in fact a reiparkably 
efficient adaptation of the CFS resource scheme. This is so because 
a node need not know hn» the directory contents will be used when 
it acquires the airectory lock. That is the uay the lock was handled 
before CFS, and praservinq this convention means that the code 
to acauire the directory lock under CFS is as efficient as possible. 
The state of the file access token, and consequently the deoree of 
sharinq of the directory contents, is determined by how the 
contents are reference^? and not by how the directory is locked. 



This means that a process may lock the directory lock without 
knowing hoy It aill reference the associated data^ and its 
reference patterns determine what other negotiations are required. 

The directory allocation table is a local "cache" for the information 
normally stored in the directory. Each active OFM is associated «ith 
a directory allocation entry. Each entry is tor exactly one directory. 
The entry, before CFS, contained: structure nuaber/ directory nu«ber# 
share count, and remainina allocation. 

Under CFS, an active allocation entry contains? structure nuaiber, 
directory nuaber, share count, and oointer to tne CFS resource block. 
The CFS resource block contains, besides the nornsal CFS control 
information, the remaining allocation for the directory and a transaction 
number. The transaction number serves the saae purpose as the transaction 
number associated with a file end-of-file oointer. 

CFS may have an "unused" resource block for a directory allocation entry. 
That is, even though there is no active directory allocation entry, 
there mav be a CItS resource block reorenentino the directory. This Is 

because CFS attempts to retain kno«iledqe of resources for as long 
as possible to avoid having to vote when sorae process Irishes to 
create the resource anew. However^ CFS yill destroy any unused resource 
allocation entry that is requested by another systeas. 

TRANSACTION f«UM&£R 

The optional data Itepis, "end-of-file oointer" and "structure free 
space", have an associated value called the "transaction nuj«ber". 

One either uses centralized or decentralized control in a "loosely-coupled 
ioultiprocessor" system. In a centralized systein, control Information and 
updating is coordinated by a master. Transactions are "serialized" by 
virtue of having a single owner for the resoruce and therefore a 
single aanager of the resource data. In a decentralized system, the 
various systems share the ownership of resources and use sowe 
sort of "concurrency control" technique to manage resources. 

CFS is a decentralized system, k resource is not owned or managed by 
any particular system, but rather the responsibility for the resource 
is passed from system to system as regulred. As such, it may not always 
be possible to uniquely identify a particular system as the owner. 
This may cause a problem when a system needs to becoae the owner, and 
therefore must determine the current status of the resource in question. 

There are t«o possibilites that a nascent owner may encounter: 

. the previous owner is present and indentif iab le. 

• There is no system that is the previous owner 

and of this latter case: 

. the existing control information is accurate 

, the existing control information in not accurate. 

Clearly, if the previous owner is oresent, tne new owner has all of 

the information it needs to oroceed with its transaction. 



If the orevious owner cannot be identified, then the ne« owner must be 
able to detecfaine ahich of the systeras has the current control 
information about the resource. It may be that none of them has, and 
this is a oroblaffl that exists even on a sinqle-processor systea. The 
result of such a problem may be "lost paqes", inconsistent data bases 
and other such phenomena. As in a sinqle-processor system, the problew 
occurs because the resource control information is lost as an effect 
of a system crashinq. 

In order to determine the wost up-to-date information about a resource* 
each system maintains a transaction count along with the information, 
whenever it acquires information with a larger transaction count than its own 
value, it knoMs that information is wore current and It must replace its own 
copy with the new data and count. Whenever a system unilaterally chanqes 
its copy of the control information, it must also increment the associated 
transaction count. Since a system may perform such an update only when 
it has write or exclusive access to the resource, the system need chanqe 
the transaction count only when it must downgrade its access. 

Due to the nature of the CFS votinq and resource wanaqeroent, it is possible 
for a system to acquire a resource but to receive a different value for 
the resource control infornatlon from each of the other svstesBs {this 
will happen only if the owner crashed. If the owner didn't crash* then 
at least two of the other systems must nave the same control information 
and transaction count). In this case, the transaction counts are used 
to identify the most up-to-date value. 

The transaction count is really a "clocfe" that is used to "tlme-stawp" 
information, ahen systems communciate with one-enother, they synchronize 
the clocks by sendinq each other the current counts. «ost network 
concurrency scheaes use clocks for similar purposes, and ipost of 
the uses and imoieraentations are considerably more exotic than this one. 
However, since CFS needs the clock only to determine relative aqes, 
and not absolute ages, of information, this simplified clock 
is adequate. 

An alternative to usinq transaction counts is to "broadcast" chanqes 

to resources. This has the disadvantaqe that it is costly in both processor 

and cownjunlcations tiiae and resources. However, CFS does use broadcastinq 

In a few cases where the lack of up-to-date Information could result 

in data beinq destroyed. The two cases are: 

. an OFM being modified and written to disk 

. an EOF value beinq written into the directory copy on the disk 

fts both of these reoresent chanqes in the oernanent copy of the resource, 
it is essential that all of the other systems have current copies 
or knowledqe of the update. 

CFS «*ESSAf^E SUMMARY 

Items marked oiita d '•■*" are sent as broadcast laessaqes. 

*1. request resource fvote) 

2. reply to request: 

a. unconaitional ye*^ 



b, unconditional no 

c, no with retry tinse 

d, conditional yes 
3» resource availaole 

4. condition satisfied 

*5. uFN updated 

*6. EOF changed 

in addition, sach message type rnay carry specific optional data Itens^ 
UP to four words of optional data oer messaqe. 



